Detecting and Analyzing the Surgical Workflow to Aid Human and Robotic Scrub Nurses

Every surgeon and nurse involved in surgery builds a mental model of each type of surgery based on their experiences. This implicit knowledge about the surgical workflow allows them to make decisions based on the current context and predict and prepare future steps. Acquiring this model takes time, therefore experienced scrub nurses are in high demand. Additionally to that, qualified scrub nurses are chronically scarce even in developed countries, which led to the development of robotic surgical assistants. In this paper we give a short introduction to surgical workflow analysis and show, how these methods can be applied to support both the education of human scrub nurses, as well as aid robotic scrub nurses. The field of surgical workflow analysis or surgical process modelling has developed in recent years with the goal to enable context-awareness in the operating room of the future [1, 2]. Current approaches to recognize the surgical process involve among other methods manual labeling [3], sensor-based instrument detection [4], evaluation of the laparoscopic video [5], and kinematic data from surgical robot systems [6]. The collected data is then usually turned into a specific model through different techniques, like Dynamic Time Warping (DTW), Hidden Markov Models (HMM), or machine learning approaches like Support Vector Machines (SVM) or random forests [4, 5, 7]. The idea for Scrub Nurse Robots (SNR) has also been in development for several years [8]. Most systems rely on voice commands from the surgeon or another nurse to reach for instruments [9, 10], but one could argue that the state models used internally for the robots are already a step towards a surgical model. Enhancing these systems through surgical phase detection and automatic predictions of upcoming instrument requests would further improve the robots performance as it could then also prepare future actions, much like experienced scrub nurses do. This paper builds on the previous work in [4], so we will apply random forests on instrument detection data in order to recognize workflow phases during a surgery. We will use this information to deduce for every step in the surgery the most probable instruments to be requested next by the main surgeon. This allows scrub nurses to prepare these instruments and provide them in time, while robotic scrub nurse systems can use this information also to optimize recognition of commands and actions from the main surgeon. Fig. 1 Binary data collected over the course of a surgery. Each instrument or mode is in use, when the corresponding line is raised.